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ABSTRACT 


The  Virtual  Space  Logistics  Readiness  Center  (VSLRC)  Living  Laboratory  was 
developed  to  provide  Air  Force  Space  Command  (AFSPC)  users  with  a  consistent 
understanding  of  space  system  operational,  equipment,  communication,  and  logistics 
status.  VSRLC  also  provides  the  ability  to  post  required  critical  briefing  materials  such  as 
Quarterly  Sustainment  Reviews,  weekly  Production  Meetings,  and  daily  Situation 
Reports  and  provide  users  with  the  most  updated  information  that  would  assist  in  data 
analysis  and  status  reporting. 

The  VSLRC  was  available  to  AFSPC  users  in  November  of  2005.  The  purpose  of 
this  Spiral  Two  research  was  to  obtain  user  feedback  on  the  effectiveness  of  VSLRC. 
Users  were  given  a  questionnaire  to  provide  feedback  after  they  had  one  month  to 
become  familiar  with  the  VSLRC  software.  Thirteen  items  of  criteria  were  established 
and  measured  through  the  questionnaire. 

The  questionnaire  results  support  the  overall  success  of  the  VSLRC  project,  as  it  met 
its  measured  criteria  in  twelve  out  of  thirteen  of  the  study’s  research  categories  including: 
the  effectiveness  of  VSLRC  visualizations,  support  for  decision  making,  providing 
situation  awareness,  perceived  satisfaction,  ease  of  learning,  ease  of  use,  improvement  in 
timeliness,  effective  and  clearly  presented  infonnation,  usefulness,  understanding  of 
reporting  functions  and  overall  usability.  Time  log  data  were  also  collected  which 
provided  a  benchmark  for  the  time  it  takes  to  update  weapon  system  status  data  and 
transfer  that  data  into  VSRLC.  It  can  be  expected  that  significant  time  savings  will  result 
when  VSRLC  is  automatically  updated  and  when  all  AFSPC  personnel  use  VSLRC  as  a 
standard  tool.  Feedback  related  to  user  ideas  for  incorporation  of  real  time  sensing 
technologies  were  solicited  and  users  indicated  several  areas  where  sensors  would  be 
useful. 
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1.  Summary 

The  Smart  Systems  for  Logistics  Command  and  Control  (SSLC2)  program  was  initiated  in 
November  2003  by  the  Air  Force  Research  Laboratory  Logistics  Readiness  Branch 
(AFRL/HEAL)  in  an  effort  to  evaluate  the  most  intelligent  use  of  technology  towards  supporting 
successful  decision  making  by  the  warfighter.  The  project’s  purpose  was  to  study  how  to  best 
fuse  current  information  sources,  using  advanced  technology  to  assess  and  present  complex  data 
to  logistics  and  operations  decision-makers  in  the  most  effective  fashion.  Additional  goals  were 
process  improvement  and  support  of  successful  collaboration. 

As  part  of  the  SSLC2  program,  the  Virtual  Space  Logistics  Readiness  Center  (VSLRC) 
Living  Laboratory  was  developed  and  delivered  in  November  2005  to  provide  Air  Force  Space 
Command  (AFSPC)  users  with  a  consistent  understanding  of  space  system  operational, 
equipment,  communication,  and  logistics  status.  Cognitive  Task  Analysis  was  accomplished 
through  interviews  with  AFSPC  users  to  storyboard  weapon  system  core  screens.  Through  user 
interface  design  and  visualization  techniques,  users  were  provided  with  an  intuitive  display  of 
information  which  requires  limited  user  training.  Each  core  screen  was  scientifically  designed 
and  developed  in  accordance  with  the  storyboards  to  include  the  capability  for  users  to  manually 
update  operational,  equipment,  and  communications  status  to  identify  impacts  of  logistics  actions 
on  operations. 

For  satellite  constellations,  operational  status  information  was  obtained  through  a  direct 
interface  with  the  Mission  Critical  Reporting  System  (MCRS).  This  demonstrated  the 
technological  ability  to  receive  near  real-time  data  interfaces  from  source  systems  which  provide 
the  user  with  automatic  status  updates,  and  created  a  baseline  for  additional  future  technological 
advances  in  intelligent  agent  alert  systems  and  additional  source  system  interfaces. 

The  VSLRC  design  and  development  efforts  also  provided  AFSPC  users  with  the  ability  to 
post  required  critical  briefing  materials  such  as  Quarterly  Sustainment  Reviews  (QSRs),  weekly 
Production  Meetings,  and  daily  Situation  Reports  (SITREPs)  to  provide  users  with  the  most 
updated  information  that  would  assist  in  data  analysis  and  status  reporting. 

The  SSLC2  Spiral  Two  is  a  research  project  sponsored  by  AFRL/HEAL  designed  to  obtain 
feedback  related  the  effectiveness  of  the  VSLRC  interface  design.  Towards  this  end,  the 
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GRACAR  team  wrote  and  administered  a  questionnaire  to  VSLRC  users,  which  was  designed  to 
measure  the  VSLRC  design  through  a  series  of  questions  in  each  of  the  following  areas: 

•  Perceived  Effectiveness  of  Visualizations 

•  Perceived  Effectiveness  for  supporting  Decision  Making 

•  Perceived  Effectiveness  for  providing  Situation  Awareness 

•  Perceived  Satisfaction  with  the  system 

•  Ease  of  Learning 

•  Timeliness  improved  by  system 

•  Information  presented  perceived  as  Effective  and  Clearly  Presented 

•  The  Functionality  is  perceived  to  be  appropriate 

•  System  is  perceived  as  Useful 

•  Reporting  functions  are  clear 

•  Easy  to  Use 

•  Overall  Perceived  System  Usability/Human  Performance  Improvement 

•  Overall  Situation  Awareness  Improvement 

Benchmarks  to  measure  success  in  each  of  these  categories  were  quantified  prior  to  the 
questionnaire  delivery  (score  of  4.0  or  better  on  a  scale  of  1  to  6).  When  tabulated,  the  Spiral 
Two  questionnaire  results  indicated  that  the  VSLRC  met  the  criteria  in  twelve  out  of  thirteen 
categories.  For  visualization,  users  gave  an  average  rating  of  5.0  (effective),  with  most  users 
indicating  the  visualizations  were  either  effective  or  extremely  effective  (6.0). 

The  one  exception  was  the  measurement  related  to  functionality,  which  resulted  in  an 
average  rating  of  3.91.  This  benchmark  was  measured  with  only  one  question  “Functionality  is 
Perceived  to  be  Appropriate”.  While  the  results  are  just  short  of  the  4.0  criteria,  the  results 
showed  that  9  of  the  1 1  respondents  gave  a  rating  of  4.0  or  higher.  Reviewing  the  user  comments 
it  appears  that  several  users  were  considering  what  VSLRC  would  have  in  future  iterations. 
Overall,  users  appear  to  be  satisfied  with  current  functionality. 

Also  included  in  the  questionnaire  was  an  open-ended  question  regarding  additional  benefits 
of  real-time  sensing  technology,  such  as  Radio  Frequency  Identification  /  Real  Time  Locating 
System  (RFID/RTLS)  for  future  integration  into  VSLRC:  “What  information  could  be  sensed 
from  these  assets  and  integrated  with  VSLRC  to  improve  your  decision  making  and  where/how 
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would  you  use  that  information?”  The  answers  collected  from  this  questionnaire  have  been 
assessed  for  future  VSLRC  capabilities  and  presented  in  Section  4.0  of  this  report. 

In  addition  to  the  questionnaire,  System  Support  Managers  (SSMs)  were  asked  to  keep  time 
logs  to  indicate  the  time  it  currently  takes  to  update  weapon  status  information  from  data 
collection  to  posting  reports  to  the  Logistics  Readiness  Center  (LRC).  The  SSMs  entered  the 
time  associated  with  each  weapon  system  over  a  period  of  ten  days.  The  results  indicated  that 
SSMs  currently  spend  an  average  of  1  hour  and  16  minutes  to  collect  the  information,  update 
PowerPoint  slides,  and  post  the  information  to  the  LRC.  After  the  information  is  posted,  one 
user  updates  the  VSLRC  to  incorporate  the  data.  It  takes  that  user  approximately  64  minutes 
each  day  to  update  the  VSLRC  and  another  30  minutes  to  create  and  post  the  daily  SITREP 
report.  On  Mondays,  which  require  additional  reports,  the  LRC  user  spends  approximately  4.15 
hours  updating  and  posting  reports.  These  times  can  be  significantly  reduced  once  automated 
reporting  is  in  place  and  when  SSMs  use  VSLRC  to  update  status  and  create  reports. 

2.  Introduction 

Logistics  support  for  the  U.S.  Military  is  a  complex,  time  dependent,  critical  task.  Creating 
Agile  Combat  Support  (ACS)  for  the  warfighter  requires  real  time  integrated  infonnation 
systems  to  support  human  decision  making.  Through  the  years,  support  tools  have  been  created 
to  support  logistics  problems.  The  SSLC2  Spiral  One  Final  Report  highlighted  some  of  these 
support  tools  and  the  importance  of  integrating  RFID  and  RTLS  information  with  additional 
legacy  systems  data  to  give  a  realistic  illustration  of  the  flightline  to  decision  makers. 

In  November  2003,  the  AFRL  initiated  the  SSLC2  Advanced  Technology  Demonstration 
(ATD)  Program  researching  technologies  and  techniques,  facilitated  through  software,  to  craft 
information  enabling  the  warfighter  to  be  more  agile,  productive  and  smart  about  critical  logistic 
resources  required  for  combat  operations.  The  Spiral  Two  research,  explored  in  this  report, 
measures  an  instantiation  of  the  SSLC2  concept  using  the  VSLRC  contract  modification  to 
SSLC2,  providing  the  AFSPC  and  Space  and  Missile  Center  (SMC)  an  actual  implementation  of 
the  SSLC2  concepts.  VSLRC  fuses  data  from  numerous  AFSPC  sources  to  provide  logistics 
status  of  space  ground  assets  to  determine  their  operational  capability.  This  research  addresses 
space  users,  providing  them  with  information  to  enhance  situational  awareness  and  helping  them 
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to  make  the  best  logistics  support  decisions  to  keep  operational,  communication,  and  equipment 
status  as  high  as  possible. 

Currently  there  is  a  need  for  consistent  understanding  of  Space  System  operational, 
equipment,  communication,  and  logistics  status.  The  VSLRC  aims  to  address  these  needs  by: 

•  Providing  an  effective  visualization  tool  to  help  AFSPC  Personnel  productively  manage 
logistics  resources  in  support  of  managing  AFSPC  ground  assets. 

•  Providing  an  effective  tool  to  assist  AFSPC  personnel  in  identifying  impacts  of  logistics 
decisions  and  actions  upon  weapon  systems  operations. 

With  these  objectives  in  mind,  the  SSLC2  team  conducted  the  Spiral  Two  research  to  obtain 
user  feedback  related  to  the  design  and  effectiveness  of  the  VSLRC  tool  for  fusing  and 
presenting  data  for  management  of  resources.  The  study  objectives  were: 

•  Evaluate  user  opinions  on  how  the  VSLRC  supports  their  tasks,  jobs  and  decisions; 

•  Obtain  user  opinions  on  the  specific  user  interface  design;  and 

•  Identify  opinions  on  improved  timeliness  of  operational  equipment  and  logistics  status. 

The  GRACAR  team  gave  VSLRC  users  a  questionnaire  to  obtain  feedback  pertaining  to  the 

study  objectives.  The  research  was  focused  on  assessing  user  opinions  of  the  VSLRC  software 
usability,  visualizations  and  improvements  to  situational  awareness  provided  by  the  new 
capabilities  introduced  into  VSLRC.  Data  was  also  collected  on  the  time  it  takes  for  AFSPC 
personnel  to  complete  their  manual  status  updating  tasks. 

2.1  PROGRAM  SCOPE 

SSCL2  is  a  6.3  ATD  program  researching  ways  to  make  the  human  warfighter  more  agile, 
productive  and  smart  about  the  critical  logistic  resources  required  to  support  combat  operations. 
SSLC2  is  an  effort  to  make  existing  systems  “smarter”  by  utilizing  data  collection  sensor 
technologies  and  fusing  data  with  existing  information  systems  to  improve  Logistics  Command 
and  Control  decision  making  and  collaboration.  The  program  produces  research,  science, 
methods,  as  well  as  the  evaluation  of  how  existing  technologies  can  be  value-added 
enhancements  to  existing/forthcoming  systems.  The  SSLC2  goal  is  to  research,  demonstrate,  and 
position  for  transition,  technology  capabilities  that  can  be  applied  to  and  integrated  into  existing 
systems  to  produce  logistics-based  operational  capability  awareness.  The  end  product  will 
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identify  requirements  and  demonstrate  technologies  that  enable  near  real  time  monitoring  of 
information  on  critical  aspects  of  the  logistics  infrastructure.  VSLRC  is  an  example  of  the 
techniques  and  tools  that  can  best  enable  logistics  decision-makers  to  have  a  high-level  view  of 
logistics  operations,  identify  potential  problems,  and  make  proactive  decisions  at  various  nodes 
within  a  logistics  support  environment  where  information  must  be  identified  and  validated. 

2.2  SSLC2  VISION  AND  SPIRALS 

The  vision  of  SSLC2  is  to  provide  decision  quality  infonnation  from  multiple  data  streams 
including  real  time  location  and  status  data  for  monitoring  capabilities  fused  with  task  and  legacy 
data  to  improve  resource  operational  capability.  The  fused  information  provides  decision-makers 
insight  into  the  operational  impact  of  their  decisions  in  advance  of  executing  the  decisions.  The 
SSLC2  effort  is  researching  the  fusion  of  technology  with  information  to  provide  decision 
support  and  situational  awareness  that  can  affect  operational  capability. 

Spiral  One  was  scoped  solely  at  wing-level  personnel  focusing  on  one  primary  decision:  the 
fix/swap  decision  of  one  aircraft.  The  goal  was  to  test  the  proof  of  concept  of  the  fusion  of 
RFID/RTLS  technology  with  legacy  data  to  provide  decision  support  and  situational  awareness. 
The  Scientific  Study  showed  that  the  user  preferred  the  integrated  decision  support  tool  that 
incorporates  RFID  and  other  logistic  infonnation  based  on  the  user  tasks  over  RFID  technology 
that  does  not,  and  over  current  practice  with  no  technology  support. 

Spiral  Two,  the  focus  of  this  report,  is  an  instantiation  of  the  SSLC2  concept  using  the 
VSLRC  contract  modification  to  SSLC2,  providing  the  AFSPC  and  SMC  an  actual 
implementation  of  the  SSLC2  concepts.  Rather  than  using  RFID/RTLS  technology  alone, 
VSLRC  fuses  data  from  numerous  AFSPC  sources  to  provide  logistics  status  of  space  ground 
assets  to  determine  their  operational  capability.  Spiral  Two  is  scoped  at  the  theater  level 
personnel  focusing  on  Space  users,  providing  fused  information  to  enhance  situational 
awareness,  and  for  making  logistics  support  decisions  to  keep  operational,  communication,  and 
equipment  status  as  high  as  possible.  Whereas  the  previous  process  was  manual  and  non¬ 
standard  across  the  space  systems,  Spiral  Two  research  focused  on  assessing  user  opinions  of  the 
visualizations  and  improvements  to  situational  awareness  provided  by  the  new  capabilities  of 
VSLRC. 
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Spiral  Three  will  expand  on  the  work  accomplished  in  Spirals  One  and  Two  by  focusing  on 
how  enhanced  data  streams  impact  complex  multiple  decisions.  Collaboration  among  users  and 
integration  of  information  is  a  key  part  of  this  Technology  Availability  Date  (TAD).  Multiple 
users  will  have  access  to  the  same  infonnation.  The  users  will  be  able  to  work  multiple  problems 
on  multiple  aircraft  at  the  same  time,  and  envision  how  decisions  affect  the  flight  line  theatre 
from  a  macro,  rather  than  micro  view.  The  technology  concept  will  be  expanded  beyond 
RFID/RTLS  location  information  to  include  additional  sensor  infonnation,  facilitation  of 
collaboration  and  enhanced  system  decision  logic.  Future  research  questions  include:  How  do 
the  visualization  techniques  affect  the  decision  making  process?  Is  the  collaboration  afforded  by 
the  system  providing  synergy?  Does  the  data  fusion  provided  by  the  system  improve  the 
timeliness  of  information  extraction  and  decision  making?  How  is  situation  awareness  (SA) 
affected?  A  key  component  of  the  research  will  be  SA,  including  the  evaluation  of  both  global 
(entire  flightline)  and  local  (current  specific  problem)  SA. 

2.3  DOCUMENT  OVERVIEW 

The  Spiral  Two  Final  Report  presents  the  method  and  results  of  a  questionnaire  designed  to 
obtain  user  feedback  related  to  the  effectiveness  of  the  VSLRC  interface  design.  It  describes  the 
various  research  activities  of  data  collection,  design,  development,  and  data  analysis  performed 
during  Spiral  Two.  The  report  is  organized  into  the  following  sections:  Section  One  gives  a  brief 
summary  of  VSLRC  Spiral  Two.  Section  Two  outlines  the  program’s  scope.  Section  Three,  the 
Methods,  Assumptions  and  Procedures  section,  highlights  the  research  objectives  and  steps  taken 
to  prepare  the  questionnaire  that  was  distributed  to  VSLRC  users.  Section  Four,  presents  an 
analysis  of  the  study  results  and  implications  for  future  work.  Section  Five,  summarizes  the 
Spiral  Two  report  given  the  study  results.  Finally,  Section  six,  Recommendations,  gives 
suggestions  for  future  work  including  possible  research  and  enhancements  related  to  the  VSLRC. 

Appendices  attached  include  a  list  of  this  document’s  Symbols,  Abbreviations,  and 
Acronyms  (Appendix  A);  the  instructions  for  completing  VSLRC  Time  Logs  and  Sample  Log 
for  both  SSMs  and  LRC  Users  (Appendix  B);  VSLRC  Questionnaire  Background  and  Read- 
Ahead  Material  (Appendix  C);  Instructions  for  Completing  the  Spiral  Two  Questionnaire 
(Appendix  D),  and  finally  the  Spiral  Two  Questionnaire  Results  and  Feedback  (Appendix  E). 
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3.  Methods,  Assumptions,  and  Procedures 

All  Spiral  Two  research  materials,  including  the  actual  questionnaire  and  instructions,  are 
included  in  the  Appendices  of  this  document.  The  following  points  detail  the  objectives  of  the 
research,  and  the  methodology  and  procedures  employed  in  its  implementation  and 
interpretation. 

3.1  RESEARCH  OBJECTIVES 

The  study  objectives  were  aligned  with  the  VSLRC  program  Key  Performance  Parameters 
(KPPs)  to: 

•  Evaluate  timeliness  of  operational  equipment  and  logistics  status  (KPP  #1) 

•  Evaluate  user  opinions  on  specific  user  interface  design  (KPP  #2) 

•  Evaluate  user  opinions  on  how  VSLRC  supports  their  tasks/job/decisions  (KPP  #3) 

The  contractor  team  gave  VSLRC  users  a  questionnaire  to  obtain  feedback  pertaining  to  the 
research  objectives.  The  research  was  focused  on  assessing  user  opinions  of  the  VSLRC  software 
usability,  visualizations  and  improvements  to  situational  awareness  and  human  perfonnance 
provided  by  this  new  capability.  The  VSLRC  aims  to  integrate  all  necessary  logistics, 
maintenance,  and  operations  data  into  a  single  system  that  decision-makers  can  use  to  identify, 
diagnose,  and  take  corrective  actions  to  solve  logistics  issues  impacting  operational  capability 
LRC  personnel  and  SSMs  were  also  asked  to  complete  a  time  log  measuring  the  time  it  currently 
takes  to  perform  data  collection  and  create  status  reports  and  how  long  it  takes  to  load  status 
information  to  the  VSLRC.  Both  of  these  data  collection  techniques  are  described  in  detail 
below. 

Participants  include  System  Support  Managers  (SSMs),  Logistics  Readiness  Center  (LRC) 
personnel,  including  Space  and  Missile  Center/Logistics  Readiness  Group  (SMC/LGX) 
personnel,  14th  AL,  ALSPC,  and  US  Strategic  Command  (USSTRATCOM)  personnel.  The  total 
number  of  solicited  participants  for  the  questionnaire  was  fifteen.  The  total  number  of 
participants  for  the  time  logs  was  six  SSMs  and  one  SMC/LGX  LRC  user. 

VSLRC  V3.0  was  released  and  implemented  on  17  November  2005.  The  GRACAR  team  e- 
mailed  the  Spiral  Two  study  read  ahead  material  two  weeks  after  the  release  of  VSLRC,  with  the 
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questionnaires  mailed  two  weeks  after  the  read-ahead.  This  allowed  participants  to  use  and 
become  familiar  with  VSRLC  for  one  month  before  they  took  the  questionnaire. 

The  GRACAR  team  asked  participants  to  return  the  questionnaire  within  one  month  from  its 
original  distribution.  GRACAR  sent  reminder  e-mails  and  offered  the  participants  the  option  to 
take  an  abbreviated  phone  interview  rather  than  complete  the  entire  questionnaire  if  they  had  not 
responded  after  two  months.  Four  of  the  participants  were  excluded.  One  participant  had 
changed  jobs  and  no  one  in  that  office  was  familiar  with  VSLRC.  One  participant  did  not  do  any 
work  related  to  VSLRC.  Another  had  seen  VSLRC  in  a  few  meetings,  but  had  never  used  it,  and 
the  final  participant  did  not  respond  to  multiple  e-mail  and  voice  mail  requests.  Of  the  eleven 
responses  obtained,  nine  completed  the  entire  questionnaire  and  two  completed  a  phone 
interview. 

During  the  phone  interview  nine  questions  were  taken  from  the  original  question  set  (see 
Appendix  E)  and  two  questions  were  added  to  summarize  their  opinions  related  to  the 
visualizations  and  situation  awareness.  These  two  questions  were  1)  The  VSLRC  visualizations 
are  effective  for  viewing  infonnation  by  weapon  system,  and  2)  VSLRC  provides  improved 
situation  awareness  of  weapon  systems. 

3.2  SPIRAL  TWO  QUESTIONNAIRE 

The  Spiral  Two  questionnaire  was  administered  to  measure  users  opinions  related  to  Human 
Perfonnance  and  Software  Usability,  Situation  Awareness,  Decision  Making,  Visualization,  and 
Sensors.  Software  Usability  refers  to  the  ease  of  learning,  ease  of  use,  usefulness,  information 
presentation  and  satisfaction  users  have  with  the  software.  Situation  Awareness  includes 
awareness  of  assets,  operational  capability,  system  health  and  the  ability  to  predict  future 
problems.  Decision  Making  incorporates  the  perceived  effectiveness  of  decision  making  support 
for  the  weapons  system,  site  and  equipment.  Visualization  measurements  evaluated  visualization 
concepts  such  as  information  grouping  and  status  indicators.  Finally,  one  question  was  asked 
about  the  usefulness  of  status  infonnation  gathered  by  sensors  to  the  space  community. 

More  specifically,  the  Spiral  Two  Research  Study  was  written  with  groups  of  questions 
specifically  designed  to  measure  the  perceived  effectiveness  of  VSLRC  in  the  following  thirteen 
areas: 
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•  Effective  Visualizations 

•  Effective  support  for  Decision  Making 

•  Effective  for  providing  Situation  Awareness 

•  User  Satisfaction  with  the  system 

•  Easy  to  Learn 

•  Timeliness  improved  by  the  system 

•  Information  presented  is  Effective  and  Clearly  Presented 

•  Functionality  is  appropriate 

•  System  is  Useful 

•  Reporting  functions  are  clear 

•  System  is  Easy  to  Use 

•  System  improves  Usability/Human  Performance 

•  System  improves  overall  Situation  Awareness 

Questions  were  crafted  to  ask  the  user  to  rank  their  level  of  agreement  with  a  positive 
statement  about  one  of  the  above  features  of  VSLRC,  or  about  the  effectiveness  of  VSLRC  in 
achieving  one  of  the  benefits.  The  questionnaire  used  two  scales;  both  ranked  1  to  6,  with  a 
selection  of  numeral  one  indicating  either  “Strongly  Disagree”  or  “Not  at  all  Effective”,  and  a 
selection  of  numeral  six  indicating  “Strongly  Agree”  or  “Extremely  Effective”.  On  a  few 
selected  questions  where  it  was  pertinent,  users  were  given  the  option:  “Not  Applicable”. 

During  questionnaire  design,  several  published  usability  questionnaires  were  evaluated. 
Questions  1-25  were  designed  by  the  team  to  address  visualization,  decision  making  and 
situation  awareness.  Questions  26-54  were  directly  from  published  usability  questionnaires 
[1,2, 3, 4], 

The  team  grouped  questions  according  to  research  topic,  and  set  a  benchmark  for  questions  at 
a  mean  of  4.0  to  draw  conclusions  for  each  grouping.  Appendix  E  of  this  document  contains  a 
table  of  the  actual  questions  used  in  the  questionnaire  showing  the  13  research  topics  coded  per 
question.  The  questionnaire  benchmark  criteria  are  shown  in  Table  1  under  the  Section  4,  Results 
and  Discussion. 

Approximately  one  month  after  the  17  November  2005  release  of  VSLRC  V3.0  the 
questionnaire  was  sent  to  participants  in  a  MS  Access  fonnat,  along  with  directions  for  how  to 
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complete  and  return  the  questionnaire.  The  graphic  below  depicts  the  Access  database  forms 
used  to  gather  questionnaire  information.  Comments  were  also  solicited  wherever  possible.  For 
the  visualization  portion  of  the  questionnaire,  a  picture  of  the  VSLRC  core  screens  was  shown. 
Users  had  the  ability  to  tailor  the  visualization  to  choose  a  specific  weapons  system  via  tabs  (See 
Figure  1  below).  Participants  moved  through  the  questionnaire  using  “next”  and  “previous” 
buttons  at  the  bottom.  Once  each  questionnaire  was  complete,  participants  chose  “finish”  and 
saved  the  file  in  an  Excel  worksheet  to  be  e-mailed  back  to  the  SSLC2  team.  The  complete  list  of 
questions  can  be  found  in  Appendix  E,  along  with  the  results. 
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Figure  1:  VSLRC  Core  Screen  Visualization 


3.3  SPIRAL  TWO  TIME  LOGS 

The  second  component  of  the  Spiral  Two  test  was  to  measure  the  time  it  takes  for  AFSPC 
personnel  to  collect,  analyze,  and  post  the  updated  status  information  on  VSLRC.  One 
participant  who  currently  inputs  all  updated  status  information  was  first  interviewed  about  the 
process.  This  process  was  used  to  create  the  time  logs  to  capture  the  desired  information.  Initial 
data  collection  with  the  LRC  user  revealed  the  following  process  for  updating  VSLRC  (see 
Figure  2). 
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Baseline  VSLRC  Posting  Process 
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Figure  2:  Processes  for  Updating  VSLRC 

At  approximately  0500  a  source  report  is  received  by  the  weapon  system  equipment 
specialist  (designated  by  the  SSM).  It  is  usually  gathered  from  different  sources  and  reported  in 
different  time  increments  (days)  depending  on  the  weapon  system.  Around  0830  each  weapon 
system  representative  takes  their  report  information,  conducts  a  teleconference  about  it  to  resolve 
discrepancies  and  translates  the  information  into  the  product  to  be  reported.  The  weapon  system 
equipment  specialist  then  converts  the  data  to  PowerPoint  to  include  locations,  color-coded 
statuses,  and  details  of  what  is  going  on.  At  approximately  1030  equipment  specialists  post  their 
PowerPoint  briefing  in  the  designated  LRC  Weapon  System  folder  by  going  to  the  LRC,  logging 
onto  the  Secure  Internet  Protocol  Network  (SIPRNET)  computer  and  updating  their  PowerPoint 
briefing  by  downloading  their  daily  file.  All  of  the  programs  usually  have  their  files  loaded  in  the 
LRC  by  1000  or  1030.  At  approximately  1130  the  LRC  representative  verifies  that  all  weapons 
system  charts  are  available  in  the  LRC.  In  the  afternoon,  the  VSLRC  is  updated  weapon  system 
by  weapon  system  to  match  the  status  report  in  the  PowerPoint  briefing. 

After  updating  the  weapon  system  information,  the  LRC  representative  generates  a  daily 
SITREP  using  the  information  from  the  Power  Point  briefings  along  with  information  from  the 
SSMs  and  14th  AF.  This  additional  data  is  usually  hot  topics  by  exception  and  comes  in  the  form 
of  web  postings,  telephones,  and  e-mails. 
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In  addition  to  this  daily  routine,  on  Mondays  four  additional  briefings  are  created  for  each 
weapons  system  -  Status,  Mission  Capability  (MICAPS),  Metrics,  and  Product  Improvements. 
These  are  posted  in  their  respected  tabs  for  each  weapon  system  on  VSLRC. 

Using  the  above  process,  team  members  created  a  time  log  to  capture  the  time  it  takes  SSMs 
to  receive  status  data,  create  their  charts,  and  post  them  in  the  LRC.  SSMs  completed  the  log  for 
ten  days  to  obtain  averages  and  this  infonnation  serves  as  a  baseline  to  understand  the  current 
process.  A  log  was  also  created  for  the  LRC  representative  to  capture  time  for  one  day  related  to 
receiving  and  posting  the  SSM  weapon  system  data  on  the  VSLRC.  The  time  log  data  entry 
sheets  can  be  found  in  Appendix  B. 

4.  Results  and  Discussion 

4.1  QUESTIONNAIRE  RESULTS 

Table  1  below  summarizes  the  results  of  the  questionnaire.  Appendix  E  provides  the 
frequency  counts  and  average  rating  for  each  individual  question.  The  benchmark  criterion  for 
all  categories  was  to  obtain  an  average  rating  of  4.0  or  higher.  As  shown  in  Table  1,  all 
categories  except  one  meet  the  benchmark.  The  question  related  to  Functionality  received  a 
3.91.  This  specific  category  is  evaluated  with  only  one  question  (question  31,  “VSLRC  has  all 
the  functions  and  capabilities  I  expect  it  to  have.”  Of  1 1  participants  1  disagreed,  2  slightly 
disagreed,  5  slightly  agreed  (rating  of  4),  and  3  agreed  (rating  of  5)  with  this  statement.  Five 
participants  provided  comments.  One  participant  commented  that  it  was  too  early  to  evaluate. 
One  participant  seemed  to  be  familiar  with  the  design  program  and  indicated  that  future 
functions  were  planned.  Three  participants  indicated  some  additional  functionality  that  they 
would  like  to  see.  It  appears  that  overall  they  are  satisfied  with  the  current  functions  available, 
but  anticipate  future  enhancements. 

Eleven  questions  were  asked  related  to  VSLRC’ s  ability  to  provide  situation  awareness 
related  to  space  weapon  systems.  The  average  response  for  these  combined  questions  meets  the 
4.0  criteria  (average  was  4.49).  It  should  be  noted  that  there  were  a  total  of  15  not  applicable 
(NA)  responses  across  these  1 1  questions.  This  may  be  because  the  concept  of  situation 
awareness  was  not  well  understood  by  the  participants.  Comments  provide  some  insight.  Five 
of  nine  participants  indicated  NA  for  the  question  “VSLRC  is  effective  in  providing  situation 
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awareness  of  Mission  Essential  Subsystem  List  (MESL)  for  each  weapon  system”.  Comments 
indicated  that  they  do  not  have  any  approved  MESL  lists.  Other  comments  indicated  that  there 
was  not  enough  time  to  have  history  data,  and  when  the  data  are  available  in  real-time,  SA  might 
be  improved.  (Specific  comments  can  be  found  in  Appendix  E.) 

Included  as  the  final  question  (number  55)  was  an  open-ended  question  regarding  additional 
benefits  of  real-time  sensing  technology  (such  as  RFID)  if  it  was  integrated  into  VSLRC.  The 
question  read:  “What  information  could  be  sensed  from  these  assets  and  integrated  with  VSLRC 
to  improve  your  decision  making  and  where/how  would  you  use  that  information?” 

Six  out  of  eleven  respondents  suggested  potential  added  future  VSLRC  real-time  sensing 
technology  features.  Among  the  comments,  it  was  suggested  users  would  benefit  from  further 
information  about  any  data  that  produces  a  change  in  the  status  of  the  system  being  reported, 
including  the  details  about  what  drove  the  status  change.  One  user  commented  that  “Real  time 
sensing  will  greatly  enhance  SA  and  decision  making”,  and  that  the  infonnation  “allows 
reports/talking  paper  to  go  up  the  chain  quicker”.  There  was  a  request  to  include  the  supply 
status,  using  sensors  on  supplies  that  are  in  forward  stocking  locations,  to  indicate  the  status  of 
the  shelf  item  when  it  was  last  tested. 

Another  user  gave  the  following  examples  of  potential  added  benefits:  “If  a  ground  antenna 
was  not  operating  and  sensors  showed  this  in  real-time  and  the  reason  was  maintenance,  this 
information  would  be  valuable.  For  ranges,  which  have  a  configuration  tree,  (we)  could  see  what 
was  mandatory  for  launch  and  what  was  broken.  Low  density/low-demand  assets  would  be 
valuable  to  track  because  they  are  something  that  are  not  used  a  lot,  therefore,  they  do  not  have  a 
demand  on  the  supply  system,  so  supply  would  not  have  spares  ready,  and  they  could  have  a 
single  point  of  failure.”  He  would  also  like  the  system  to  sense  and  show  when  radoms  blow  off 
in  high  winds.  User  comments  have  been  noted,  and  will  be  evaluated  for  inclusion  in  future 
VSLRC  releases  or  enhancements.  All  user  comments  can  be  found  in  Appendix  E. 


13 


Table  1:  Questionnaire  Results 


Code 

Goal 

Metric  Goal 

Actual 

Mean 

Standard 

Deviation 

VIS 

VSLRC  visualizations  are  effective. 

Mean  score  of  4.0  or  higher  on  questions  1-9  combined. 

5.06 

0.50 

EDM 

Perceived  Effectiveness  for  supporting 

decision  making. 

Mean  score  of  4.0  or  higher  on  questions  10-14  combined. 

4.61 

0.58 

ESA 

Perceived  Effectiveness  for  providing 

Situation  Awareness 

Mean  score  of  4.0  or  higher  on  questions  15-25  combined. 

4.49 

0.70 

SAT 

Perceived  satisfaction  with  the 
system. 

Mean  score  of  4.0  or  higher  on  questions  44  and  45  combined. 

4.75 

0.72 

ETL 

System  should  be  perceived  as  easy 
to  learn 

Mean  score  of  4.0  or  higher  on  questions  26  and  38  combined. 

4.61 

0.61 

TIM 

System  should  improve  timeliness 

Mean  score  of  4.0  or  higher  on  question  27.  (KPP) 

4.00 

0.71 

ECP 

Information  presented  is  perceived  as 

effective  and  clearly  presented 

Mean  score  of  4.0  or  higher  on  questions  28-30  combined. 

4.44 

0.58 

FUN 

The  functionality  is  perceived  to  be 
appropriate. 

Mean  score  of  4.0  or  higher  on  question  31 . 

3.91 

0.94 

USE 

System  should  be  perceived  as  useful 

Mean  score  of  4.0  or  higher  on  questions  32-37  combined  (Davis, 
1989) 

4.14 

0.82 

REP 

Understanding  reporting  functions 

Mean  score  of  4.0  or  higher  on  questions  46-48. 

4.37 

0.93 

EOU 

System  should  be  perceived  as  easy 
to  use 

Mean  score  of  4.0  or  higher  on  questions  38-43  combined  (Davis, 
1989) 

4.66 

0.79 

UUI 

Overall  Perceived  System  Usability/ 
Human  Performance 

Mean  score  of  4.0  or  higher  on  all  23  usability  questions  (26-48) 
and  on  questions  49-51. 

4.41 

0.78 

OSA 

Overall  Situation  Awareness 

Mean  score  of  4.0  or  higher  on  questions  52-54. 

4.22 

0.69 

Overall 

Collapsed  across  all  questions 

4.44 

0.72 
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4.2  TIME  LOG  RESULTS 

As  described  in  section  3.3  SSMs  collect  data,  update  slides  and  then  post  the  information  to 
the  LRC.  The  times  associated  with  this  process  are  shown  in  Table  2. 

Table  2:  Average  times  reported  by  SSM  by  weapon  system  and  task 

Average  time  is  in  minutes;  N  is  number  of  data  points. _ _ 


Weapon  System 

Collect  Information 

Update  Slides 

Update  LRC 

GPST 

15 

N=  10 

10 

N=  10 

05 

N=  10 

LTRS 

103 

N=  7 

30 

N=  1 

28 

N=  4 

SBIRS 

17 

N=  10 

20 

N=  10 

15 

N=  10 

AFSCN 

17 

N=  10 

62 

N=  10 

13 

N=  10 

DMSP 

No  Data 

No  Data 

05 

N=  10 

MilSatCom 

73 

N=  3 

08 

N=3 

13 

N=  3 

Averaged  Total  Time 

36 

(N=40) 

29 

(N=34) 

11 

(N=47) 

Average  time  is  in  minutes;  N  is  number  of  data  points. 

On  average  it  takes  approximately  1  hour  and  16  minutes  to  collect  the  information,  update 
PowerPoint  slides,  and  post  the  information  to  LRC.  The  most  time  is  spent  collecting  the 
information  (average  of  36  minutes).  The  standard  deviation  is  large  because  data  collection 
times  for  LTRS  and  MilSatCom  take  the  most  time.  For  Air  Force  Satellite  Control  Network 
(AFSCN)  it  took  more  time  to  update  the  slides  (an  average  of  1  hour)  than  to  collect  the 
information.  Posting  the  infonnation  to  LRC  took  on  average  1 1  minutes,  with  the  most  time 
spent  updating  LTRS. 

Once  the  data  are  posted  to  the  LRC,  the  LRC  representative  downloads  the  data  and 
manually  updates  the  VSLRC.  Only  one  user  updates  the  VSLRC.  The  user  sent  back  a  log  but 
did  not  provide  their  start  and  end  times.  Instead  they  gave  their  estimate  of  duration  in  minutes; 
therefore,  the  data  are  not  necessarily  accurate.  The  data  provided  showed  that  the  LRC 
representative  updated  13  separate  weapon  systems  and  it  took  an  average  of  4.92  minutes,  with 
a  maximum  time  of  1 5  minutes  and  a  minimum  time  of  2  minutes,  and  a  standard  deviation  of 
4.21  minutes.  The  total  time  to  update  all  weapon  systems  was  estimated  to  be  64  minutes.  After 
the  information  is  updated  for  each  weapon  system,  the  LRC  representative  spends  another  30 
minutes  to  create  and  post  the  daily  SITREP  report. 

In  addition  to  the  daily  tasks  described  above,  the  LRC  representative  creates  four  additional 
briefings  for  each  weapon  system.  The  time  reported  was  5  minutes  for  each  weapon  system  for 
a  total  of  65  minutes.  Additionally,  90  minutes  are  spent  creating  Monday  meeting  charts  for  the 
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production  meeting.  Therefore,  on  Monday  the  LRC  representative  spends  a  total  of  249 
minutes  (4.15  hrs)  to  update  weapons  system  information  and  create  charts. 

It  is  obvious  from  these  data  that  updating  weapon  systems  information  is  time  consuming. 
The  concept  of  VSLRC  is  to  provide  the  opportunity  for  automatic  updating  and  standardization 
in  reporting.  If  all  AFSPC  personnel  were  using  VSLRC  to  post  information  it  would  eliminate 
the  need  for  the  SSM  to  create  PowerPoint  slides  and  update  them  to  the  LCR  and  it  would 
eliminate  all  the  tasks  conducted  by  the  LRC  representative  because  all  users  would  have  access 
to  the  same  data.  Additionally,  when  the  data  can  be  automatically  input  into  VSLRC,  the  SSM 
would  not  need  to  input  the  data,  only  verify  it.  A  test  was  conducted  under  the  VSLRC  for 
satellite  constellations  only  showing  that  VSLRC  can  be  tied  directly  to  MCRS  for  automated 
updating. 

5.  Conclusions 

VSLRC  is  a  very  important  and  useful  tool  for  AFSPC.  The  results  of  this  study  indicate  that 
users  are  satisfied  with  the  VSLRC  visualizations  and  interactions,  it  is  a  useable  tool  that  is  easy 
to  leam  and  effective  for  providing  situation  awareness  and  for  supporting  decision  making. 
Users  would  like  additional  functionality  and  suggested  future  enhancements  including  real  time 
sensors  that  can  provide  automatic  updates  and  reports  related  to  each  weapon  system  to  reduce 
the  time  spent  collecting  data,  creating  slides,  and  posting  information.  The  weapon  system 
specialist  would  still  need  to  do  some  verification  and  resolving  of  discrepancies,  but  could 
spend  more  time  on  other  critical  tasks.  More  in-depth  usability  testing  should  be  conducted  in 
order  to  provide  input  to  future  design  enhancements. 

6.  Recommendations 

The  Spiral  Two  VSLRC  research  project  demonstrates  the  value  added  of  the  VSLRC  to 
space  users.  When  VSLRC  is  used  by  all  AFSPC  personnel  as  a  standard  reporting  and  status 
tool,  the  benefits  will  increase.  The  results  for  this  study  are  subjective,  and  a  usability  study  that 
measures  objective  perfonnance  and  the  impact  of  this  system  on  logistics  decisions  should  be 
conducted  to  determine  the  most  beneficial  updates.  Investigations  of  possible  enhancements  to 
VSLRC  are  also  listed  below.  Some  of  these  were  reported  in  the  VSLRC  Living  Laboratory 
Final  Report  and  some  were  collected  during  this  study. 
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•  Add  additional  information  gathered  through  remote  real  time  sensing  and  determine 
how  to  fuse  this  infonnation  with  existing  visualizations. 

•  Incorporate  role-based  alerts  that  will  notify  users  that  a  condition  exists  that  may 
require  immediate  or  fast  attention,  and  are  meant  to  inform  as  well  as  promote  action. 

•  Display  performance  measures  and  trends  over  time  to  indicate  the  health  of  each 
system,  potential  short-  and  long-term  problems,  and  ways  of  taking  corrective  action. 

•  Development  of  business  rules  related  to  Operations  Capability,  Systems  Capability, 
Fully-Mission  Capable,  Partially-Mission  Capable,  and  Non-Mission  Capable  statuses 
such  that  VSLRC  can  automatically  calculate  and  identify  appropriate  conditions  and 
report  to  AFSPC  upper  echelons. 

•  Collection  of  maintenance  and  logistics  data  that  support  metrics  calculation. 

•  Provide  the  ability  for  users  to  drill-down  from  equipment  status  to  underlying  causes. 

•  Track  and  store  actions  being  taken  to  solve  logistics  problems  as  they  pertain  to 
systems,  components,  items,  etc. 

•  Information/diagnostic  screens  that  integrate  data  from  multiple  Air  Force,  other  DoD, 
and  contractor  legacy  infonnation  and  analytical  systems. 

•  Three-dimensional  map  view  to  show  precise  longitude  and  latitude  (or  GPS 
coordinates)  coordinates  for  each  location  with  day/night  shading. 

•  Ability  to  bi-directionally  transfer  infonnation  across  security  domains  with  the 
utilization  of  a  guard  processor. 

•  Ability  to  utilize  a  collaborative  workspace  and  “chat”  capability  within  the  VSLRC 
resident  on  the  SIPRNET. 

•  Ability  to  see  impacts  of  logistics  and  maintenance  decisions  on  operational  readiness. 

•  Incorporation  of  real  time  legacy  data  systems  for  real  time  or  near  real  time  update  of 
status  (E.g.,  Government  owned  data  sources  such  as  Core  Automated  Maintenance 
System,  Air  Force  Knowledge  System,  Standard  Base  Supply  System,  etc.) 
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Acronym 

Definition 

ACS 

Agile  Combat  Support 

AFRL/HEAL 

Air  Force  Research  Laboratory/  Warfighter  Readiness  Research  Division/ 

Logistics  Readiness  Branch 

AFSCN 

Air  Force  Satellite  Control  Network 

AFSPC 

Air  Force  Space  Command 

ATD 

Applied  Technology  Demonstration 

DMSP 

Defense  Meteorological  Satellite  Program 

GPST 

Global  Positioning  System 

KPP 

Key  Performance  Parameter 

LRC 

Logistics  Readiness  Center 

MESL 

Mission  Essential  Subsystem  List 

MICAP 

Mission  Capability 

MCRS 

Mission  Critical  Reporting  System 

QSR 

Quarterly  Sustainment  Review 

RFID 

Radio  Frequency  Identification 

RTLS 

Real-Time  Locating  System 

SA 

Situation  Awareness 

SBIRS 

Space  Based  Infrared  System 

SIPRNET 

Secure  Internet  Protocol  Network 

SITREP 

Situation  Report 

SMC 

Space  and  Missile  Center 

SMC/LGX 

Space  and  Missile  Center/Logistics  Readiness  Group 

SSLC2 

Smart  Systems  for  Logistics  Command  and  Control 

SSM 

System  Support  Managers 

TAD 

Technology  Availability  date 

USSTRATCOM 

United  States  Strategic  Command 

VSLRC 

Virtual  Space  Logistics  Readiness  Center 
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System  Support  Manager: 


The  GRACAR  contracting  team  is  requesting  your  support  for  providing  feedback  on  the 
Virtual  Space  Logistics  Readiness  Center  (VSLRC).  VSLRC  provides  the  Air  Force  Space 
Command  (AFSPC)  and  Space  and  Missile  Systems  Center  (SMC)  logistics  status  of  space 
ground  assets  to  determine  their  operational  capability.  This  program  includes  gaining  a 
thorough  understanding  of  the  Space  unique  decision  making  processes,  determining  the  logistics 
impacts  on  space  operations,  and  providing  “most-current”  logistics  data  in  “actionable” 
decision-quality  format  to  VSLRC  system  users.  Your  input  is  crucial  to  ensure  that  the 
application  and  further  development  of  this  technology  is  designed  to  meet  your  needs. 

The  purpose  of  this  short  time  log  is  to  evaluate  how  the  information  in  VSLRC  improves  the 
timeliness  of  work  tasks  and  decisions.  System  Support  Manager  Representatives  are  asked  to 
enter  the  time  associated  with  each  of  the  activities  stated  below  within  the  next  five  workdays  as 
they  complete  Program  Status  Updates.  Upon  completion  of  the  log  please  fax  this  sheet  to  937- 
281-6701,  Attn:  Smart  Systems. 

Thank  you  in  advance  for  agreeing  to  complete  the  log.  If  you  have  any  questions  about  the 
study  please  contact  the  GRACAR  team  at  e.matthews@gracar.com. 
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System  Manager  User  Log:  One  Time  Log 


Instructions:  Please  enter  the  hours  and  minutes  associated  with  each  of  the  activities  stated  below. 

This  information  is  to  be  used  for  future  design  improvements  to  the  VSLRC 

Date: 

Program:  (e.g.),  AFSCN,  SSN,  etc.) 

Start  Time 

End  Time 

Enter  time  required  to  collect  information  for  Program  Status  Update  (e.g.,  all 

information  necessary  to  make  changes  to  PowerPoint  slides).  Include  time 

required  for  all  activities  including  collecting  reports,  teleconferences,  web 

searches,  etc. 

Enter  time  required  to  update  your  slides  based  on  the  known  status  of  your 

system.  This  is  before  you  take  the  updates  to  the  LRC. 

Enter  time  required  to  take  status  information  (e.g.,  your  slides)  to  the  LRC 

and  update  them  on  the  LRC  computer. 

PLEASE  FAX  THIS  COMPLETED  FORM  TO  937-281-6701 
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LRC  VSLRC  User: 


The  GRACAR  contracting  team  is  requesting  your  support  for  providing  feedback  on  the 
Virtual  Space  Logistics  Readiness  Center  (VSLRC).  VSLRC  provides  the  Air  Force  Space 
Command  (AFSPC)  and  Space  and  Missile  Systems  Center  (SMC)  logistics  status  of  space 
ground  assets  to  determine  their  operational  capability.  This  program  includes  gaining  a 
thorough  understanding  of  the  Space  unique  decision  making  processes,  determining  the  logistics 
impacts  on  space  operations,  and  providing  “most-current”  logistics  data  in  “actionable” 
decision-quality  fonnat  to  VSLRC  system  users.  Your  input  is  crucial  to  ensure  that  the 
application  and  further  development  of  this  technology  is  designed  to  meet  your  needs. 

The  purpose  of  this  time  log  is  to  evaluate  how  the  information  in  VSLRC  improves  the 
timeliness  of  updating  status  information  for  each  weapons  system.  You  are  asked  to  enter  the 
time  associated  with  each  of  the  activities  stated  below  for  a  period  of  ten  workdays.  Upon 
completion  of  these  logs  please  fax  to  937-281-6701,  Attn;  Smart  Systems. 

Thank  you  in  advance  for  agreeing  to  complete  the  log.  If  you  have  any  questions  about  the 
study  please  contact  the  GRACAR  team  at  e.matthews@gracar.com. 
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VSLRC  LRC  User  Log:  Daily  Log 


Instructions:  Please  enter  the  hours  and  minutes  associated  with  each  of  the  activities  stated  below.  This 
information  is  to  be  used  for  future  design  improvements  to  the  VSLRC. 

Time  Modified  or 

_  Posted  ..  *  ■  Product 

Program  .  ,  __ .  _  *  Status  MICAP  Metric 

in  LRC  by  System  Improveme 


AFSCN 

Start  Time 

End  Time 

DMSP  -  MKIVB/RSS 

Start  Time 

End  Time 

DMSP-SEON 

Start  Time 

End  Time 

DMSP-IMS 

Start  Time 

End  Time 

DMSP-SBIRS 

Start  Time 

End  Time 

EWR 

Start  Time 

End  Time 

GPS 

Start  Time 

End  Time 

mii  qatpom  - 

Start  Time 

IVIILOn  1  WIVI  -  U0L>0 

End  Time 

MILSATCOM  -  MILSTAR 

Start  Time 

End  Time 

MILSATCOM  -  SCTS 

Start  Time 

End  Time 

Spacelift  Range  -  East 

Start  Time 

End  Time 

Spacelift  Range  -  West 

Start  Time 

End  Time 

SSN 

Start  Time 

End  Time 

Enter  time  required  to 
create  and  post  the  daily 

Start  Time 

SITREP  on  the  VSLRC: 

End  Time 

Enter  time  required  to 
create  the  Monday 
meeting  charts: 

Start  Time 

End  Time 
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VSLRC  Questionnaire  Background  and  Read-Ahead  Material 


The  GRACAR  contracting  team  is  requesting  your  support  for  providing  feedback  on  the 
Virtual  Space  Logistics  Readiness  Center  (VSLRC).  VSLRC  provides  the  Air  Force  Space 
Command  (AFSPC)  and  Space  and  Missile  Systems  Center  (SMC)  logistics  status  of  space 
ground  assets  to  determine  their  operational  capability.  This  program  includes  gaining  a 
thorough  understanding  of  the  Space  unique  decision  making  processes,  determining  the  logistics 
impacts  on  space  operations,  and  providing  “most-current”  logistics  data  in  “actionable” 
decision-quality  format  to  VSLRC  system  users.  Your  input  is  crucial  to  ensure  that  the 
application  and  further  development  of  this  technology  is  designed  to  meet  your  needs.  To  obtain 
feedback  on  the  technology  and  visualizations  used  in  VSLRC  the  GRACAR  contractor  team 
will  be  sending  a  questionnaire  to  VSLRC  users. 

Thank  you  in  advance  for  agreeing  to  participate  in  this  study.  As  a  participant  you  will 
be  asked  to  complete  a  questionnaire  assessing  the  visualization,  decision  support,  usability, 
situational  awareness  and  performance  support  that  VSLRC  provides.  The  GRACAR  contractor 
team  will  e-mail  you  a  questionnaire  that  was  created  in  MS  ACCESS  and  instructions  on  how  to 
complete  the  questionnaire.  Please  complete  the  questionnaire  within  four  weeks.  We  will  send 
a  reminder  e-mail  approximately  two  weeks  after  the  questionnaire  has  been  sent  if  necessary. 

If  you  have  any  questions  about  the  study  or  have  trouble  obtaining  the  questionnaire 
once  it  is  sent  please  contact  the  GRACAR  team  at  e.matthews@gracar.com. 
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Spiral  Two  Questionnaire  Instructions 
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QUESTIONNAIRE  INSTRUCTIONS 


Listed  below  are  the  instructions  for  extracting  and  taking  the  questionnaire.  If  you  have  any 
questions  about  the  study  or  have  trouble  opening  the  questionnaire,  please  contact  the 
GRACAR  team  at  e.matthews@gracar.com. 


1.  Click  on  the  questionnaire  attachment.  You  will  be  asked  “Do  you  want  to  open  or  save  this 
file?”  Choose  Open.  It  may  take  a  couple  seconds  to  open  the  file. 


File  Download 


Do  you  want  to  open  01  save  this  file? 


n 


Name:  NewSurveys.zip 
Type:  Compressed  (zipped)  Folder 
From:  65.54.229.250 


|  Open  ]  [  Save  |  [  Cancel  ] 


While  files  from  the  Internet  can  be  useful,  some  files  can  potentially 
W«7  harm  your  computer.  If  you  do  not  trust  the  source,  do  not  open  or 
save  this  file.  What's  the  risk? 


2.  Once  the  document  has  been  downloaded  you  will  see  the  following  window.  Choose 

“Extract  all  files.” 


3.  The  Extraction  Wizard  will  pop  up.  Choose  Next.  On  the  next  screen  it  will  tell  you  that  the 
file  will  be  saved  in  temporary  internet  files.  Select  Next.  It  will  take  a  couple  seconds  to  extract 
the  file.  You  will  then  see  the  “Extraction  Complete”  window.  Make  sure  the  box  “Show 
extracted  files”  is  checked  and  choose  Finish. 


4.  A  new  window  will  pop  up  with  the  Questionnaires  file  icon.  Click  on  “Spiral2Questionaire”. 
If  you  see  a  security  warning  and  ask  you  if  you  want  to  open  the  file,  click  Open.  Choose  Open 
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again  if  you  see  another  security  warning.  The  security  warning  is  caused  of  scripts  that  were 
written  in  the  Access  database  to  make  it  more  user  friendly.  None  of  the  files  associated  with 
this  questionnaire  are  harmful  to  you  computer. 

Note:  If  the  questionnaire  will  not  open  or  if  it  does  open  but  gives  you  an  error  message  and  is 
not  editable  try  re-running  step  3  but  change  the  path.  This  may  allow  you  to  use  the 
questionnaire. 

5.  The  VSLRC  Questionnaire  will  appear  at  Question  1.  Use  the  maximize  screen  icon  in  the  top 
right  corner  of  the  window  to  make  sure  the  questionnaire  is  open  to  full  screen.  This  allows  you 
to  see  all  the  selections. 


Please  note  that  the  questionnaire  includes  a  graphic  representation  of  VSLRC  (see  the  picture 
below),  which  is  intended  to  assist  you  in  remembering  the  layout  of  the  actual  program.  Please 
answer  the  questionnaire  questions  in  relation  to  your  experience  with  VSLRC  itself,  and  not  the 
graphics  in  the  questionnaire. 

Complete  the  survey  by  answering  each  question  using  the  scale  at  the  top.  Each  question  has  a 
space  giving  you  the  option  to  add  any  additional  comments.  We  welcome  your  comments 
because  they  provide  us  with  additional  feedback  that  helps  us  better  design  and  improve 
systems  for  you  as  a  user. 


Use  the  scroll  bar  at  the  right  to  view  the  entire  page  if  necessary.  Once  a  question  is  complete  go 
to  the  next  question  using  the  next  button.  To  change  the  graphic  to  reflect  the  view  you  are 
evaluating,  choose  from  the  weapons  system  tabs  above  the  VSLRC  picture. 


Question 


Weapons 
System  Tabs 


>  »  6»w»* 


C*  »  *32'.  *1 


MMnMtoii 


k>«  4  (**(»•«  *- Nat *»<»-»  1wrt>tlK»n  [be**  [bow 

i e  *  «  m  mu 


Scroll 

^Bar 


I  UtrWv'figttto  M&SArCCM  Xxitfn  »»v"  X*.* 


> 

Ex  F*T  I  a 

mm 

Next  Button 


6.  After  completing  question  55,  choose  Finish.  This  will  save  your  questionnaire  answers  in  an 
Excel  file  named  “Survey”  which  you  will  find  in  the  “My  Documents”  folder  on  your  computer. 
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Please  e-mail  this  Excel  file  to  e.matthews@gracar.com.  If  you  have  any  questions  about  the 
study  or  have  trouble  opening  the  questionnaire,  please  contact  the  GRACAR  team  at 
e.matthews@gracar.com. 
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Spiral  Two  Questionnaire 
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VSLRC  Questionnaire 


The  questionnaire  is  listed  below.  The  frequency  counts  for  answers  to  each  question  are  listed  under  each  answer.  The  average 
score  for  each  question  is  listed  in  the  last  column.  The  questions  marked  with  an  **  were  the  questions  used  in  the  phone  interview 
for  two  participants.  Therefore,  N  =  1 1  for  these  questions. 

Instructions  to  User:  Please  provide  a  rating  to  each  question  below  that  best  matches  your  rating  of  effectiveness  as  it  relates  to 
VSLRC. 


CODE 

Statements 

Not  at  all 
Effective 

1 

Not 

Effective 

2 

Somewhat 

Not 

Effective 

3 

Somewhat 

Effective 

4 

Effective 

5 

Extremely 

Effective 

6 

NA 

Average 
Score 
(Std  Dev) 

VIS 

1 .  The  visualization  and  selection  of 
weapon  systems  by  organization  is: 

1 

6 

2 

5.11 

(0.61) 

VIS 

2.  The  visualization  and  selection  of 
weapon  systems  by  type  is: 

8 

1 

5.11 

(0.33) 

VIS 

3.  The  visualization  and  selection  of 
information  by  site  is: 

7 

2 

5.22 

(0.44) 

VIS 

4.  The  visualization  of  status 

indicators  directly  above  related 
equipment  icons  is: 

7 

2 

5.22 

(0.44) 

VIS 

5.  The  visualization  of  geographic 
references  are: 

7 

2 

5.22 

(0.44) 

VIS 

6.  The  visualization  of  status 

indicators  (yellow,  green,  red)  are: 

2 

5 

2 

5.0 

(7.1) 

VIS 

7.  The  equipment  status  icons, 
including  ETRO  days,  are: 

1 

7 

1 

5.0 

(0.50) 
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CODE 

Statements 

Not  at  all 
Effective 

1 

Not 

Effective 

2 

Somewhat 

Not 

Effective 

3 

Somewhat 

Effective 

4 

Effective 

5 

Extremely 

Effective 

6 

NA 

Average 
Score 
(Std  Dev) 

VIS 

8.  Labels  on  status  indicators, 
equipment,  etc.  are: 

3 

6 

4.67 

(0.50) 

VIS** 

9.  Overall,  the  visualization  of  the 
weapons  system  view  is: 

1 

9 

1 

5.0 

(0.33) 

EDM 

10.  With  regard  to  supporting  my 

decision  making  process,  the  ability 
to  see  the  status  of  all  weapons 
systems  is: 

2 

7 

4.78 

(0.44) 

EDM 

1 1 .  With  regard  to  supporting  my 
decision  making  process  for  a 
weapons  system,  the  information 
provided  is: 

3 

6 

4.67 

(0.50) 

EDM 

12.  With  regard  to  supporting  my 
decision  making  process  for  any 
given  site,  the  information  provided 
is: 

2 

7 

4.78 

(0.44) 

EDM 

13.  With  regard  to  supporting  my 
decision  making  process  for  any 
given  piece  of  equipment,  the 
information  provided  is: 

5 

4 

4.44 

(0.53) 

EDM** 

14.  With  regard  to  supporting  my 
decision  making  process,  VSLRC 
is: 

2 

2 

6 

1 

4.4 

(0.84) 
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The  following  graphic  points  out  the  sections  of  the  VSLRC  visualization  that  each  of  questions  one  through  eight  pertains  to: 
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The  following  12  questions  ask  the  user  about  the  ability  for  VSLRC  to  provide  them  with  situation  awareness  related  to  space 
weapon  systems.  The  term  situation  awareness  can  be  described  broadly  as  a  person’s  state  of  knowledge  of  the  situation  around  him 
or  her.  The  user  was  instructed  to  please  rate  VSLRC  on  it’s  effectiveness  for  providing  situation  awareness  on  the  items  listed  in  the 
questions  below. 

Please  rate  the  effectiveness  of  VSLRC  for  providing  the  following: 


CODE 

Statement 

Not  at  all 
Effective 

1 

Not 

Effective 

2 

Somewhat 

Not 

Effective 

3 

Somewhat 

Effective 

4 

Effective 

5 

Extremely 

Effective 

6 

NA 

Average 
Score 
(Std  Dev) 

ESA 

15. 

VSLRC  is  effective  in  providing 
situation  awareness  of  status  of 
single  equipment  components. 

4 

4 

1 

4.67 

(0.71) 

ESA 

16. 

VSLRC  is  effective  in  providing 
situation  awareness  of 
operations  capability  of  each 
weapon  system. 

6 

3 

4.33 

(0.50) 

ESA 

17. 

VSLRC  is  effective  in  providing 
situation  awareness  of  the  space 
asset  organizations  and  their 
locations. 

4 

4 

1 

4.67 

(0.71) 

ESA 

18. 

VSLRC  is  effective  in  providing 
situation  awareness  of  estimated 
times  to  complete  equipment 
maintenance  or  upgrades 

5 

3 

1 

4.55 

(0.73) 

ESA 

19. 

VSLRC  is  effective  in  providing 
situation  awareness  of  overall 
weapon  system  health  status. 

5 

4 

4.44 

(0.53) 

ESA 

20. 

VSLRC  is  effective  in  providing 
situation  awareness  of  weather 
status. 

1 

1 

4 

3 

4.33 

(0.21) 

ESA 

21. 

VSLRC  is  effective  in  providing 
situation  awareness  of  weapon 
systems  status  history. 

1 

2 

4 

2 

4.43 

(0.79) 
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CODE 

Statement 

Not  at  all 
Effective 

1 

Not 

Effective 

2 

Somewhat 

Not 

Effective 

3 

Somewhat 

Effective 

4 

Effective 

5 

Extremely 

Effective 

6 

NA 

Average 
Score 
(Std  Dev) 

ESA 

22. 

VSLRC  is  effective  in  providing 
situation  awareness  of  single 
equipment  component  status 
history 

1 

1 

5 

2 

4.57 

(0.79) 

ESA 

23. 

VSLRC  is  effective  in  providing 
situation  awareness  of  MESL  for 
each  weapon  system. 

1 

2 

1 

5 

4.0 

(0.81) 

ESA 

24. 

VSLRC  is  effective  in  providing 
situation  awareness  of 
communication  capability  of  each 
weapon  system. 

1 

3 

4 

1 

4.37 

(0.74) 

ESA 

25. 

VSLRC  is  effective  in  providing 
situation  awareness  of  location 
specific  site  status  (active,  hot, 
warm,  no  status) 

2 

5 

2 

4.71 

(0.49) 

Please  rate  your  level  of  agreement  with  the  following  statements. 

CODE 

Question 

Strongly 

Disagree 

1 

Disagree 

2 

Slightly 

Disagree 

3 

Slightly 

Agree 

4 

Agree 

5 

Strongly 

Agree 

6 

Average 
Score 
(Std  Dev) 

UUI/ 

ETL 

26. 

1  believe  1  became  more 
productive  quickly  using  VSLRC 

6 

3 

4.33 

(0.50) 

UUI/ 

TIM 

27. 

It  takes  less  time  to  make 
decisions  with  regard  to 
problems  within  a  weapon 
system 

2 

5 

2 

4.0 

(0.71) 

UUI/ 

ECP 

28. 

It  is  easy  to  find  the  information  1 
needed 

6 

3 

4.33 

(0.50) 

UUI/ 

ECP** 

29. 

The  information  is  effective  in 
helping  me  complete  my  tasks 

1 

6 

4 

4.27 

(0.65) 
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CODE 

Question 

Strongly 

Disagree 

1 

Disagree 

2 

Slightly 

Disagree 

3 

Slightly 

Agree 

4 

Agree 

5 

Strongly 

Agree 

6 

Average 
Score 
(Std  Dev) 

UUI/ 

ECP 

30. 

The  organization  of  the  menus 
and  information  lists  seems  quite 
logical  and  useful 

3 

6 

4.67 

(0.50) 

UUI/ 

FUN** 

31. 

VSLRC  has  all  the  functions  and 
capabilities  1  expect  it  to  have 

1 

2 

5 

3 

3.91 

(0.94) 

UUI/ 

USE 

32. 

Using  VSLRC  in  my  job  enables 
me  to  accomplish  tasks  more 
quickly. 

1 

2 

3 

3 

3.89 

(1.05) 

UUI/ 

USE 

33. 

Using  VSLRC  improves  my  job 
performance 

1 

1 

4 

3 

4.0 

(1.0) 

UUI/ 

USE** 

34. 

Using  VSLRC  increases  my 
productivity 

2 

5 

4 

4.18 

(0.75) 

UUI/ 

USE 

35. 

Using  VSLRC  enhances  my 
effectiveness  on  the  job 

1 

5 

3 

4.18 

(0.87) 

UUI/ 

USE 

36. 

Using  VSLRC  makes  it  easier  to 
do  my  job 

6 

3 

4.33 

(0.50) 

UUI/ 

USE 

37. 

1  have  found  VSLRC  useful  in  my 
job 

1 

4 

4 

4.33 

(0.71) 

UUI/ 

ETLV 

EOU 

38. 

Learning  to  operate  VSLRC  was 
easy  for  me 

2 

6 

1 

4.89 

(0.61) 

UUI/ 

EOU 

39. 

1  find  it  easy  to  get  VSLRC  to  do 
what  1  want  it  to  do 

1 

3 

5 

4.44 

(0.73) 

UUI/ 

EOU 

40. 

1  find  my  interaction  with  VSLRC 
to  be  clear  and  understandable 

1 

2 

6 

4.55 

(0.72) 

UUI/E 

OU 

41. 

1  find  VSLRC  to  be  flexible  to 
interact  with 

1 

3 

4 

1 

4.55 

(0.88) 

UUI/ 

EOU 

42. 

It  was  easy  for  me  to  become 
skillful  at  using  VSLRC. 

1 

3 

4 

1 

4.55 

(0.88) 

IIU/ 

EOU** 

43. 

1  find  VSLRC  easy  to  use 

1 

2 

4 

3 

4.91 

(0.94) 
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CODE 

Question 

Strongly 

Disagree 

1 

Disagree 

2 

Slightly 

Disagree 

3 

Slightly 

Agree 

4 

Agree 

5 

Strongly 

Agree 

6 

Average 
Score 
(Std  Dev) 

UUI/ 

SAT 

44. 

1  feel  comfortable  using  VSLRC 

3 

5 

1 

4.78 

(0.67) 

UUI/ 

SAT** 

45. 

Overall,  1  am  satisfied  with 

VSLRC 

1 

2 

7 

1 

4.72 

(0.79) 

UUI/ 

REP 

46. 

It  is  easy  to  post  reports. 

3 

4 

1 

1 

4.0 

(10) 

UUI/ 

REP 

47. 

It  is  easy  to  view  posted  reports. 

1 

8 

4.78 

(0.67) 

UUI/ 

REP 

48. 

It  is  easy  to  download  posted 
reports. 

3 

6 

4.33 

(1.0) 

UUI 

49. 

The  technologies  provided  are 
usable 

3 

6 

4.67 

(0.50) 

UUI 

50. 

1  am  provided  with  useful 
decision  support 

5 

4 

4.44 

(0.53) 

UUI** 

51. 

This  technology  positively 
impacts  the  visibility  of  status  of 
space  logistics  assets 

4 

7 

4.64 

(0.50) 

OSA 

52. 

It  is  easy  to  notice  when  there 
are  problems  with  equipment 

3 

6 

4.67 

(0.50) 

OSA 

53. 

When  problems  occur,  it  is  easy 
to  understand  the  problem 

2 

4 

3 

4.11 

(0.78) 

OSA 

54. 

When  problems  occur,  1  am  able 
to  predict  when  the  problem  will 
be  fixed 

2 

6 

1 

3.89 

(0.60) 
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55.  Incorporation  of  real  time  sensing  technology  can  improve  logistics  support.  Sensors  can  be  applied  to  AFSPC  assets  to  provide  information 
such  as  the  following: 

•  Location 

•  Status  (is  it  broken,  in  use,  not  in  use) 

•  Battery  level 

•  Fluid  levels 

•  Temperatures 

•  Vibration 

•  Etc. 

What  information  could  be  sensed  from  these  assets  and  integrated  with  VSLRC  to  improve  your  decision  making  and  where/how  would  you  use 
that  information? 


Comments: 
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Spiral  Two  Questionnaire  Feedback  Comments 


User  comments  by  question  are  presented  below  and  grouped  by  category. 


Category 

Question 

Comment 

Visualization 

Question  6:  The  visualization  of  status  indicators 
(yellow,  green,  red)  are: 

•  What  metric  is  used  to  determine  red,  yellow,  green?  Define  the  metric 
used  for  determining  status. 

Question  8:  Labels  on  status  indicators,  equipment, 
etc.  are 

•  Going  to  the  Help  menu  for  the  Acronym  List  is  time  consuming.  An 
Acronym  Tab  could  eliminate  the  need  to  go  extra  steps. 

•  Need  to  have  a  "key/legend"  explaining  what  each  acronym  stands  for. 

Question  9:  Overall,  the  visualization  of  the  weapons 
system  view  is 

•  Need  to  have  a  "key/legend"  explaining  what  each  acronym  stands  for. 

•  Most  users  are  pretty  happy,  especially  since  they  helped  design  the 
screens 

Decision  Making 

Question  10:  With  regard  to  supporting  my  decision 
making  process,  the  ability  to  see  the  status  of  all 
weapons  systems  is: 

•  Is  this  system  the  tool  used  by  all  for  status?  If  so  it  becomes  more 
effective. 

Question  11:  With  regard  to  supporting  my  decision 
making  process  for  a  weapons  system,  the 
information  provided  is: 

•  Is  this  system  the  tool  used  by  all  for  status?  If  so  it  becomes  more 
effective. 

Question  12:  With  regard  to  supporting  my  decision 
making  process  for  any  given  site,  the  information 
provided  is: 

•  Is  this  system  the  tool  used  by  all  for  status,  if  so  it  becomes  more 
effective? 

Question  14:  With  regard  to  supporting  my  decision 
making  process,  VSLRC  is: 

•  1  believe  we  are  still  in  a  trial  period,  using  the  VSLRC  in  the  decision 

making  process  will  require  it  to  become  the  standard  for  all 
reports/briefings 

Situational 

Awareness 

Question:  VSLRC  provides  improved  situation 
awareness  of  weapon  systems 

•  Right  now. 

•  Based  on  the  current  state  of  development 

Question  15:  VSLRC  is  effective  in  providing 
situation  awareness  of  status  of  single  equipment 
components. 

•  Needs  more  detail  info  in  write-up 

Question  17:  VSLRC  is  effective  in  providing 
situation  awareness  of  the  space  asset  organizations 
and  their  locations. 

•  SA  of  organizations  was  not  really  part  of  the  design. 

Question  18:  VSLRC  is  effective  in  providing 
situation  awareness  of  estimated  times  to  complete 
equipment  maintenance  or  upgrades. 

•  Will  be  very  effective  if  detailed  information  is  given.  Don't  make  people 
have  to  guess  about  what's  being  reported. 

•  Needs  to  be  as  specific  as  possible 
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Category 

Question 

Comment 

Question  19:  VSLRC  is  effective  in  providing 
situation  awareness  of  overall  weapon  system  health 
status. 

•  Will  be  more  effective  if  information  is  detailed  and  up-to-date. 

•  The  criteria  for  health  is  still  pending. 

Question  20:  VSLRC  is  effective  in  providing 
situation  awareness  of  weather  status. 

•  Never  tried  it 

•  Weather  is  usually  not  updated  for  the  sites 

Question  21:  VSLRC  is  effective  in  providing 
situation  awareness  of  weapon  system  status  history. 

•  Time  will  enable  the  history  to  contain  relevant  information 

•  Maybe  after  it's  been  used  for  a  period  of  time. 

•  Never  tried  it. 

•  Not  seen 

Question  22:  VSLRC  is  effective  in  providing 
situation  awareness  of  single  equipment  component 
status  history. 

•  Time  will  enable  the  history  to  contain  relevant  information 

•  Maybe  after  it's  been  used  for  a  period  of  time. 

•  Having  just  one  window  for  historical  tracking  versus  a  current  one  then 
the  history  will  enable  users  to  get  to  the  info  easier 

•  Never  tried  it 

•  Didn't  see  the  detail  on  this 

Question  23:  VSLRC  is  effective  in  providing 
situation  awareness  of  MESL  for  each  weapon 
system. 

•  Have  not  seen  any  references  to  MESL,  don't  know  if  1  am  missing  it  or  it 
is  used  in  the  background 

•  Approved  MESL  does  not  currently  exist 

•  We  do  not  have  approved  MESLs. 

•  VSLRC  is  not  as  detailed  as  other  sources  for  the  MESL  for  each  system 

Question  24:  VSLRC  is  effective  in  providing 
situation  awareness  of  communication  capability  of 
each  weapon  system. 

•  Will  be  very  effective  if  detailed,  up-to-date  information  is  displayed 

•  ????? 

•  This  is  hard  to  define. 

•  Effective  if  it  highlights  comm  systems  used 

•  VSLRC  provides  comm  status  but  not  capability  since  it  does  not  provide 
operational  impacts 

Question  25:  VSLRC  is  effective  in  providing 
situation  awareness  of  location  specific  site  status 
(active,  hot,  warm,  no  status). 

•  Do  not  understand  the  reference 

•  Yes  but  not  in  terms  of  real  or  near  real  time. 

Timeliness 

Question  27:  It  takes  less  time  to  make  decisions 
with  regard  to  problems  within  a  weapon  system. 

•  Sometimes  1  still  have  to  refer  to  the  SITREP  for  additional  information 

•  If  the  information  is  detailed  and  up-to-date,  then  yes. 

•  Because  1  needed  to  get  outside  info  not  detailed  in  status 

Effective 

Presentation 

Question  28:  It  is  easy  to  find  the  information  1  need. 

•  A  detailed  listing  of  all  equipment  at  each  location  would  be  helpful 

•  I'm  not  too  familiar  with  the  program  so  it  is  currently  taking  some  time  to 
find  the  information.  This  will  improve  with  time  and  program  familiarity. 

•  Some  of  the  information  would  be  hard  to  find  for  a  new  user  since  the 
information  is  in  different  locations  i.e.  MICAPS,  Production  slides,  etc. 

Question  29:  The  information  is  effective  in  helping 
me  complete  my  tasks. 

•  A  detailed  listing  of  all  equipment  at  each  location  would  be  helpful 
.  N/A 
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Category 

Question 

Comment 

Functionality 

Question  31 :  VSLRC  has  all  the  functions  and 
capabilities  1  expect  it  to  have. 

•  The  VSLRC  is  still  not  the  norm,  when  this  becomes  true,  functions  and 
capabilities  can  better  evaluated 

•  A  detailed  listing  of  all  equipment  at  each  location  would  be  helpful 

•  Links  to  14AF  Newsgroup  could  add  more  fidelity  on  outages 

•  Additional  functions/capabilities  are  needed/planned. 

•  VSLRC  needs  to  be  made  so  that  we  can  place  multiple  write  ups  for 
sites  in  different  windows.  Right  now  you  have  to  clump  them  together  in 
one  window 

•  Some  functionality  has  not  yet  been  developed 

Usefulness 

Question  32:  Using  VSLRC  in  my  job  enables  me  to 
accomplish  tasks  more  quickly. 

•  Not  always  sure  of  the  accuracy  and  currency  of  the  data,  additional 
sources  are  sometimes  needed  for  confirmation  of  info. 

•  If  a  hot  SIPR  connection  was  on  my  desk  it  would  be. 

Question  33:  Using  VSLRC  improves  my  job 
performance. 

•  Only  for  high-level  discussions. 

Question  34:  Using  VSLRC  increases  my 

productivity 

•  It  will,  once  it  becomes  more  established  and  is  populated  with  all  the 
information  1  need.  A  detailed  list  of  equipment  at  each  location. 

Question  35:  Using  VSLRC  enhances  my 

effectiveness  on  the  job 

•  Only  because  it  has  access  to  the  status  briefings  that  speak  about 
financial  issues. 

Question  36:  Using  VSLRC  makes  it  easier  to  do  my 
job. 

•  Only  because  it  has  access  to  the  status  briefings  that  speak  about 
financial  issues. 

Question  37:  1  have  found  VSLRC  useful  in  my  job. 

•  It  will,  once  it  becomes  more  established  and  is  populated  with  all  the 
information  1  need.  A  detailed  list  of  equipment  at  each  location 

•  Only  because  it  has  access  to  the  status  briefings  that  speak  about 
financial  issues 

Ease  of  Use 

Question  39:  1  find  it  easy  to  get  VSLRC  to  do  what  1 
want  it  to  do. 

•  Still  not  sure  of  all  the  capabilities 

•  It  will,  once  it  becomes  more  established  and  is  populated  with  all  the 
information  1  need.  A  detailed  list  of  equipment  at  each  location. 

Question  42:  It  was  easy  for  me  to  become  skillful  at 
using  VSLRC. 

•  Not  sure  1  know  of  all  the  functions/capabilities  of  VSLRC 

Satisfaction 

Question  45:  Overall,  1  am  satisfied  with  VSLRC. 

•  It  will  be  more  useful  once  it  becomes  more  established  and  is  populated 
with  all  the  information  1  need.  A  detailed  list  of  equipment  at  each 
location. 

•  Needs  to  have  more  info  and  links  to  show  history  and  status  of 
equipment 

Reporting 

Question  46:  It  is  easy  to  post  reports. 

•  Never  had  to  post  reports,  no  comment  not  available 

•  Don't  know  yet 

•  Never  used 

Question  47:  It  is  easy  to  view  posted  reports 

•  No  comment 

Question  48:  It  is  easy  to  download  posted  reports. 

•  No  comment 

•  Don't  know 

•  Never  downloaded. 
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Category 

Question 

Comment 

Usability 

Question  49:  The  technologies  provided  are  usable. 

•  Better  to  pull  information  for  this  system  from  CAMS 

Question  51:  This  technology  positively  impacts  the 
visibility  of  status  of  space  logistics  assets. 

•  It  will,  once  it  becomes  more  established  and  is  populated  with  all  the 
information  1  need.  A  detailed  list  of  equipment  at  each  location. 

Overall  Situational 
Awareness 

Question  53:  When  problems  occur,  it  is  easy  to 
understand  the  problem. 

•  What  kind  of  problems? 

•  If  detailed  information  is  given. 

•  Never  happened 

•  Usually  have  to  verify  with  other,  more  detailed  information 

Question  54:  When  problems  occur,  1  am  able  to 
predict  when  the  problem  will  be  fixed. 

•  What  kind  of  problems? 

•  Only  if  detailed,  correct  and  up-to-date  information  is  given 

•  Never  happened  to  me.  Don't  use  it  extensively  enough. 

Sensors 

Question  55:  Incorporation  of  real  time  sensing 
technology  can  improve  logistics  support.  Sensors 
can  be  applied  to  AFSPC  assets  to  provide 
information  such  as  the  following:  Location,  Status 
(is  it  broken,  in  use,  not  in  use),  Battery  level,  Fluid 
levels,  Temp 

•  Any  information  that  produces  a  change  in  the  status  of  the  system 
should  be  reported  and  the  information  that  drove  the  status  change 
should  also  be  available. 

•  Real  time  sensing  will  greatly  enhanced  SA  and  decision  making.  This 
info  will  allow  reports/talking  paper  to  go  up  the  chain  quicker. 

•  Budget  available  versus  budget  needed  for  the  execution  and  out  years. 

•  Tie  into  CAMS  and  comm  systems 

•  Space  is  not  as  dynamic  as  Air.  Maybe  for  the  supply  status  with  items  in 
forward  stocking  locations;  such  as  the  last  tested  status  of  items  sitting 
on  the  shelf. 

•  Some  examples:  If  a  ground  antenna  was  not  operating  and  sensors 
showed  this  in  real-time  and  the  reason  was  maintenance,  this 
information  would  be  valuable.  For  ranges,  which  have  a  configuration 
tree,  if  they  could  see  what  was  mandatory  for  launch  and  what  was 
broken.  He  also  thinks  low  density/low-demand  assets  would  be  valuable 
to  track  because  they  are  something  that  are  not  used  a  lot;  therefore, 
they  do  not  have  a  demand  on  the  supply  system,  so  supply  would  not 
have  spares  ready,  and  they  could  have  a  single  point  failure.  He  would 
like  to  be  able  to  see  when  radoms  blow  off  in  high  winds. 
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